{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMaker LLM-as-Judge Evaluation - Basic Usage\n",
    "\n",
    "This notebook demonstrates the basic user-facing flow for creating and managing LLM-as-Judge evaluation jobs using the LLMAsJudgeEvaluator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configuration\n",
    "REGION = 'us-west-2'\n",
    "S3_BUCKET = 's3://mufi-test-serverless-smtj/eval/'\n",
    "# DATASET = 'arn:aws:sagemaker:us-west-2:052150106756:hub-content/AIRegistry/DataSet/gen-qa-test-content/1.0.1'  # Dataset ARN or S3 URI\n",
    "DATASET = \"s3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-dataset/gen_qa.jsonl\"\n",
    "MLFLOW_ARN = 'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Import Required Libraries\n",
    "\n",
    "Import the LLMAsJudgeEvaluator class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.train.evaluate import LLMAsJudgeEvaluator\n",
    "from rich.pretty import pprint\n",
    "\n",
    "# Configure logging to show INFO messages\n",
    "import logging\n",
    "logging.basicConfig(\n",
    "    level=logging.INFO,\n",
    "    format='%(levelname)s - %(name)s - %(message)s'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create LLMAsJudgeEvaluator\n",
    "\n",
    "Create an LLMAsJudgeEvaluator instance with the desired evaluator model, dataset, and metrics.\n",
    "\n",
    "### Key Parameters:\n",
    "- `model`: Model Package (or Base Model) to be evaluated (required)\n",
    "- `evaluator_model`: Bedrock model ID to use as judge (required)\n",
    "- `dataset`: S3 URI or Dataset ARN (required)\n",
    "- `builtin_metrics`: List of built-in metrics (optional, no 'Builtin.' prefix needed)\n",
    "- `custom_metrics`: JSON string of custom metrics (optional)\n",
    "- `evaluate_base_model`: Whether to evaluate base model in addition to custom model (optional, default=True)\n",
    "- `mlflow_resource_arn`: MLflow tracking server ARN (optional)\n",
    "- `model_package_group`: Model package group ARN (optional)\n",
    "- `s3_output_path`: S3 output location (required)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Using custom metrics (as JSON string)\n",
    "\n",
    "Custom metrics must be provided as a properly escaped JSON string. You can either:\n",
    "1. Create a Python dict and use `json.dumps()` to convert it\n",
    "2. Provide a pre-escaped JSON string directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Method 1: Create dict and convert to JSON string\n",
    "custom_metric_dict = {\n",
    "    \"customMetricDefinition\": {\n",
    "        \"name\": \"PositiveSentiment\",\n",
    "        \"instructions\": (\n",
    "            \"You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. \"\n",
    "            \"Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\\n\\n\"\n",
    "            \"Consider the following:\\n\"\n",
    "            \"- Does the response have a positive, encouraging tone?\\n\"\n",
    "            \"- Is the response helpful and constructive?\\n\"\n",
    "            \"- Does it avoid negative language or criticism?\\n\\n\"\n",
    "            \"Rate on this scale:\\n\"\n",
    "            \"- Good: Response has positive sentiment\\n\"\n",
    "            \"- Poor: Response lacks positive sentiment\\n\\n\"\n",
    "            \"Here is the actual task:\\n\"\n",
    "            \"Prompt: {{prompt}}\\n\"\n",
    "            \"Response: {{prediction}}\"\n",
    "        ),\n",
    "        \"ratingScale\": [\n",
    "            {\"definition\": \"Good\", \"value\": {\"floatValue\": 1}},\n",
    "            {\"definition\": \"Poor\", \"value\": {\"floatValue\": 0}}\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "# Convert to JSON string\n",
    "custom_metrics_json = json.dumps([custom_metric_dict])  # Note: wrap in list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:43:52] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found credentials in shared credentials file: ~<span style=\"color: #e100e1; text-decoration-color: #e100e1\">/.aws/credentials</span>   <a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">credentials.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py#1364\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1364</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:43:52]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found credentials in shared credentials file: ~\u001b[38;2;225;0;225m/.aws/\u001b[0m\u001b[38;2;225;0;225mcredentials\u001b[0m   \u001b]8;id=406523;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py\u001b\\\u001b[2mcredentials.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=534480;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py#1364\u001b\\\u001b[2m1364\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /Users/mufi/Library/Application Support/sagemaker/config.yaml\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved MLflow resource ARN:                                    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#113\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">113</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         mmlu-eval-experiment                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved MLflow resource ARN:                                    \u001b]8;id=360312;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=805617;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#113\u001b\\\u001b[2m113\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/ \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         mmlu-eval-experiment                                             \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">LLMAsJudgeEvaluator</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">region</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">sagemaker_session</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">sagemaker.core.helper.session_helper.Session</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x15f5c11c0</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">base_eval_name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'eval-meta-04295d90'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_resource_arn</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_experiment_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_run_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">networking</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">kms_key_id</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model_package_group</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">evaluator_model</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'anthropic.claude-3-5-haiku-20241022-v1:0'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">dataset</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-dataset/gen_qa.jsonl'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">builtin_metrics</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008700; text-decoration-color: #008700\">'Completeness'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'Faithfulness'</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">custom_metrics</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">[{</span><span style=\"color: #008700; text-decoration-color: #008700\">\"customMetricDefinition\": </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">\"name\": \"PositiveSentiment\", \"instructions\": \"You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\\\\n\\\\nConsider the following:\\\\n- Does the response have a positive, encouraging tone?\\\\n- Is the response helpful and constructive?\\\\n- Does it avoid negative language or criticism?\\\\n\\\\nRate on this scale:\\\\n- Good: Response has positive sentiment\\\\n- Poor: Response lacks positive sentiment\\\\n\\\\nHere is the actual task:\\\\nPrompt: </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">{{</span><span style=\"color: #008700; text-decoration-color: #008700\">prompt</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">}}</span><span style=\"color: #008700; text-decoration-color: #008700\">\\\\nResponse: </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">{{</span><span style=\"color: #008700; text-decoration-color: #008700\">prediction</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">}}</span><span style=\"color: #008700; text-decoration-color: #008700\">\", \"ratingScale\": </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">[{</span><span style=\"color: #008700; text-decoration-color: #008700\">\"definition\": \"Good\", \"value\": </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">\"floatValue\": 1</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">}}</span><span style=\"color: #008700; text-decoration-color: #008700\">, </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">\"definition\": \"Poor\", \"value\": </span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">\"floatValue\": 0</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">}}]}}]</span><span style=\"color: #008700; text-decoration-color: #008700\">'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">evaluate_base_model</span>=<span style=\"color: #d70000; text-decoration-color: #d70000; font-style: italic\">False</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mLLMAsJudgeEvaluator\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mregion\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msagemaker_session\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225msagemaker.core.helper.session_helper.Session\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x15f5c11c0\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbase_eval_name\u001b[0m=\u001b[38;2;0;135;0m'eval-meta-04295d90'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m=\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_resource_arn\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_experiment_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_run_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mnetworking\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mkms_key_id\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel_package_group\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mevaluator_model\u001b[0m=\u001b[38;2;0;135;0m'anthropic.claude-3-5-haiku-20241022-v1:0'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mdataset\u001b[0m=\u001b[38;2;0;135;0m's3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-dataset/gen_qa.jsonl'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbuiltin_metrics\u001b[0m=\u001b[1m[\u001b[0m\u001b[38;2;0;135;0m'Completeness'\u001b[0m, \u001b[38;2;0;135;0m'Faithfulness'\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mcustom_metrics\u001b[0m=\u001b[38;2;0;135;0m'\u001b[0m\u001b[1;38;2;0;135;0m[\u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0m\"customMetricDefinition\": \u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0m\"name\": \"PositiveSentiment\", \"instructions\": \"You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\\\\n\\\\nConsider the following:\\\\n- Does the response have a positive, encouraging tone?\\\\n- Is the response helpful and constructive?\\\\n- Does it avoid negative language or criticism?\\\\n\\\\nRate on this scale:\\\\n- Good: Response has positive sentiment\\\\n- Poor: Response lacks positive sentiment\\\\n\\\\nHere is the actual task:\\\\nPrompt: \u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0mprompt\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[38;2;0;135;0m\\\\nResponse: \u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0mprediction\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[38;2;0;135;0m\", \"ratingScale\": \u001b[0m\u001b[1;38;2;0;135;0m[\u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0m\"definition\": \"Good\", \"value\": \u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0m\"floatValue\": 1\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[38;2;0;135;0m, \u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0m\"definition\": \"Poor\", \"value\": \u001b[0m\u001b[1;38;2;0;135;0m{\u001b[0m\u001b[38;2;0;135;0m\"floatValue\": 0\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m]\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m}\u001b[0m\u001b[1;38;2;0;135;0m]\u001b[0m\u001b[38;2;0;135;0m'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mevaluate_base_model\u001b[0m=\u001b[3;38;2;215;0;0mFalse\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# Create evaluator with custom metrics\n",
    "evaluator = LLMAsJudgeEvaluator(\n",
    "    # base_model='arn:aws:sagemaker:us-west-2:052150106756:model-package/Demo-test-deb-2/1',  # Required\n",
    "    model=\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\",\n",
    "    evaluator_model=\"anthropic.claude-3-5-haiku-20241022-v1:0\",  # Required\n",
    "    dataset=DATASET,  # Required: S3 URI or Dataset ARN\n",
    "    builtin_metrics=[\"Completeness\", \"Faithfulness\"],  # Optional: Can combine with custom metrics\n",
    "    custom_metrics=custom_metrics_json,  # Optional: JSON string of custom metrics\n",
    "    mlflow_resource_arn=MLFLOW_ARN,  # Optional\n",
    "    # model_package_group=MODEL_PACKAGE_GROUP,  # Optional if BASE_MODEL is a Model Package ARN/Object\n",
    "    s3_output_path=S3_BUCKET,  # Required\n",
    "    evaluate_base_model=False\n",
    ")\n",
    "\n",
    "pprint(evaluator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Optional] Example with multiple custom metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Create multiple custom metrics\n",
    "# custom_metrics_list = [\n",
    "#     {\n",
    "#         \"customMetricDefinition\": {\n",
    "#             \"name\": \"GoodMetric\",\n",
    "#             \"instructions\": (\n",
    "#                 \"Assess if the response has positive sentiment. \"\n",
    "#                 \"Prompt: {{prompt}}\\nResponse: {{prediction}}\"\n",
    "#             ),\n",
    "#             \"ratingScale\": [\n",
    "#                 {\"definition\": \"Good\", \"value\": {\"floatValue\": 1}},\n",
    "#                 {\"definition\": \"Poor\", \"value\": {\"floatValue\": 0}}\n",
    "#             ]\n",
    "#         }\n",
    "#     },\n",
    "#     {\n",
    "#         \"customMetricDefinition\": {\n",
    "#             \"name\": \"BadMetric\",\n",
    "#             \"instructions\": (\n",
    "#                 \"Assess if the response has negative sentiment. \"\n",
    "#                 \"Prompt: {{prompt}}\\nResponse: {{prediction}}\"\n",
    "#             ),\n",
    "#             \"ratingScale\": [\n",
    "#                 {\"definition\": \"Bad\", \"value\": {\"floatValue\": 1}},\n",
    "#                 {\"definition\": \"Good\", \"value\": {\"floatValue\": 0}}\n",
    "#             ]\n",
    "#         }\n",
    "#     }\n",
    "# ]\n",
    "\n",
    "# # Convert list to JSON string\n",
    "# custom_metrics_json = json.dumps(custom_metrics_list)\n",
    "\n",
    "# # Create evaluator\n",
    "# evaluator = LLMAsJudgeEvaluator(\n",
    "#     base_model=BASE_MODEL,\n",
    "#     evaluator_model=\"anthropic.claude-3-5-haiku-20241022-v1:0\",\n",
    "#     dataset=DATASET,\n",
    "#     custom_metrics=custom_metrics_json,  # Multiple custom metrics\n",
    "#     s3_output_path=S3_BUCKET,\n",
    "# )\n",
    "\n",
    "# print(f\"✅ Created evaluator with {len(json.loads(custom_metrics_json))} custom metrics\")\n",
    "# pprint(evaluator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Optional] Skipping base model evaluation (evaluate custom model only)\n",
    "\n",
    "By default, LLM-as-Judge evaluates both the base model and custom model. You can skip base model evaluation to save time and cost by setting `evaluate_base_model=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Define custom metrics (same as test script)\n",
    "# custom_metrics = \"[{\\\"customMetricDefinition\\\":{\\\"name\\\":\\\"GoodMetric\\\",\\\"instructions\\\":\\\"You are an expert evaluator. Your task is to assess if the sentiment of the response is positive. Rate the response based on whether it conveys positive sentiment, helpfulness, and constructive tone.\\\\n\\\\nConsider the following:\\\\n- Does the response have a positive, encouraging tone?\\\\n- Is the response helpful and constructive?\\\\n- Does it avoid negative language or criticism?\\\\n\\\\nRate on this scale:\\\\n- Good: Response has positive sentiment\\\\n- Poor: Response lacks positive sentiment\\\\n\\\\nHere is the actual task:\\\\nPrompt: {{prompt}}\\\\nResponse: {{prediction}}\\\",\\\"ratingScale\\\":[{\\\"definition\\\":\\\"Good\\\",\\\"value\\\":{\\\"floatValue\\\":1}},{\\\"definition\\\":\\\"Poor\\\",\\\"value\\\":{\\\"floatValue\\\":0}}]}},{\\\"customMetricDefinition\\\":{\\\"name\\\":\\\"BadMetric\\\",\\\"instructions\\\":\\\"You are an expert evaluator. Your task is to assess if the sentiment of the response is negative. Rate the response based on whether it conveys negative sentiment, unhelpfulness, or destructive tone.\\\\n\\\\nConsider the following:\\\\n- Does the response have a negative, discouraging tone?\\\\n- Is the response unhelpful or destructive?\\\\n- Does it use negative language or harsh criticism?\\\\n\\\\nRate on this scale:\\\\n- Bad: Response has negative sentiment\\\\n- Good: Response lacks negative sentiment\\\\n\\\\nHere is the actual task:\\\\nPrompt: {{prompt}}\\\\nResponse: {{prediction}}\\\",\\\"ratingScale\\\":[{\\\"definition\\\":\\\"Bad\\\",\\\"value\\\":{\\\"floatValue\\\":1}},{\\\"definition\\\":\\\"Good\\\",\\\"value\\\":{\\\"floatValue\\\":0}}]}}]\"\n",
    "\n",
    "# # Create evaluator that only evaluates the custom model (matching test script exactly)\n",
    "# evaluator = LLMAsJudgeEvaluator(\n",
    "#     base_model=BASE_MODEL,\n",
    "#     evaluator_model=\"anthropic.claude-3-5-haiku-20241022-v1:0\",\n",
    "#     dataset=DATASET,\n",
    "#     builtin_metrics=[\"Completeness\", \"Faithfulness\", \"Helpfulness\"],\n",
    "#     custom_metrics=custom_metrics,\n",
    "#     mlflow_resource_arn=MLFLOW_ARN,\n",
    "#     model_package_group=MODEL_PACKAGE_GROUP,\n",
    "#     model_artifact=MODEL_ARTIFACT,\n",
    "#     s3_output_path=S3_BUCKET,\n",
    "#     evaluate_base_model=False,  # KEY: Skip base model evaluation\n",
    "# )\n",
    "\n",
    "# print(\"✅ Created evaluator (custom model only)\")\n",
    "# pprint(evaluator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Run LLM-as-Judge Evaluation\n",
    "\n",
    "Start the evaluation job. The evaluator will:\n",
    "1. Generate inference responses from the base model (if evaluate_base_model=True)\n",
    "2. Generate inference responses from the custom model\n",
    "3. Use the judge model to evaluate responses with built-in and custom metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:22:01] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:22:01]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=931878;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=760856;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Getting or creating artifact for source:                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#597\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">597</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Getting or creating artifact for source:                         \u001b]8;id=179503;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=71430;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#597\u001b\\\u001b[2m597\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for existing artifact for model package:               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#459\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">459</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for existing artifact for model package:               \u001b]8;id=2444;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=787547;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#459\u001b\\\u001b[2m459\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found existing artifact:                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#468\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">468</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         138877d772ec489bef                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found existing artifact:                                         \u001b]8;id=808361;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=665812;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#468\u001b\\\u001b[2m468\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         138877d772ec489bef                                               \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Inferred model package group ARN:                                <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#386\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">386</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         t-finetuned-models-gamma from                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Inferred model package group ARN:                                \u001b]8;id=361400;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=518747;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#386\u001b\\\u001b[2m386\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         t-finetuned-models-gamma from                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Automatically inferred model_package_group:                      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#421\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">421</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         t-finetuned-models-gamma                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Automatically inferred model_package_group:                      \u001b]8;id=299761;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=867866;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#421\u001b\\\u001b[2m421\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         t-finetuned-models-gamma                                         \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using ModelPackage - model_package_group_arn:            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">llm_as_judge_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#319\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">319</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-g <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         roup/test-finetuned-models-gamma                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using ModelPackage - model_package_group_arn:            \u001b]8;id=538256;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\u001b\\\u001b[2mllm_as_judge_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=292230;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#319\u001b\\\u001b[2m319\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-g \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         roup/test-finetuned-models-gamma                         \u001b[2m                             \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved model info - base_model_name:                   <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">llm_as_judge_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#322\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">322</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct,               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         base_model_arn:                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPub <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         licHub/Model/meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">.10</span>.<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, source_model_package_arn:                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/t <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         est-finetuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved model info - base_model_name:                   \u001b]8;id=854970;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\u001b\\\u001b[2mllm_as_judge_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=553794;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#322\u001b\\\u001b[2m322\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct,               \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         base_model_arn:                                          \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPub \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         licHub/Model/meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct/\u001b[1;36m1\u001b[0m \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1;36m.10\u001b[0m.\u001b[1;36m0\u001b[0m, source_model_package_arn:                         \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/t \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         est-finetuned-models-gamma/\u001b[1;36m28\u001b[0m                            \u001b[2m                             \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Uploading custom metrics to S3:                          <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">llm_as_judge_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#220\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">220</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/evaluationinputs/eva</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">l-meta-04295d9020251130-002201/custom-metrics.json</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Uploading custom metrics to S3:                          \u001b]8;id=657021;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\u001b\\\u001b[2mllm_as_judge_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=5404;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#220\u001b\\\u001b[2m220\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/evaluationinputs/eva\u001b[0m \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225ml-meta-04295d9020251130-002201/\u001b[0m\u001b[38;2;225;0;225mcustom-metrics.json\u001b[0m       \u001b[2m                             \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Successfully uploaded custom metrics to:                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">llm_as_judge_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#228\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">228</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/evaluationinputs/eva</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">l-meta-04295d9020251130-002201/custom-metrics.json</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                             </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Successfully uploaded custom metrics to:                 \u001b]8;id=718083;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py\u001b\\\u001b[2mllm_as_judge_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=581773;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/llm_as_judge_evaluator.py#228\u001b\\\u001b[2m228\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/evaluationinputs/eva\u001b[0m \u001b[2m                             \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225ml-meta-04295d9020251130-002201/\u001b[0m\u001b[38;2;225;0;225mcustom-metrics.json\u001b[0m       \u001b[2m                             \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using full template for ModelPackage                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#655\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">655</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using full template for ModelPackage                             \u001b]8;id=143249;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=489338;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#655\u001b\\\u001b[2m655\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved template parameters: <span style=\"font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">'role_arn'</span>:                       <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#693\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">693</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:iam::052150106756:role/Admin'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_resource_arn'</span>:   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">/mmlu-eval-experiment'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_experiment_name'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_run_name'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'model_package_group_arn'</span>:              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">st-finetuned-models-gamma'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'source_model_package_arn'</span>:          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'base_model_arn'</span>:                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0'</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3_output_path'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval'</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'dataset_artifact_arn'</span>:                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">3138877d772ec489bef'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'action_arn_prefix'</span>:                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:action'</span>,               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'dataset_uri'</span>:                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-datas</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">et/gen_qa.jsonl'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'judge_model_id'</span>:                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'anthropic.claude-3-5-haiku-20241022-v1:0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'llmaj_metrics'</span>:     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">[</span><span style=\"color: #008700; text-decoration-color: #008700\">\"Completeness\", \"Faithfulness\"</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">]</span><span style=\"color: #008700; text-decoration-color: #008700\">'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'custom_metrics_s3_path'</span>:    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/evaluationinputs/eval-meta-</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">04295d9020251130-002201/custom-metrics.json'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>:  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>,       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'pipeline_name'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'SagemakerModelEvaluationType2-llmaj'</span>,          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'evaluate_base_model'</span>: <span style=\"color: #d70000; text-decoration-color: #d70000; font-style: italic\">False</span><span style=\"font-weight: bold\">}</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved template parameters: \u001b[1m{\u001b[0m\u001b[38;2;0;135;0m'role_arn'\u001b[0m:                       \u001b]8;id=109479;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=566018;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#693\u001b\\\u001b[2m693\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:iam::052150106756:role/Admin'\u001b[0m, \u001b[38;2;0;135;0m'mlflow_resource_arn'\u001b[0m:   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m/mmlu-eval-experiment'\u001b[0m, \u001b[38;2;0;135;0m'mlflow_experiment_name'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m,          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'mlflow_run_name'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m, \u001b[38;2;0;135;0m'model_package_group_arn'\u001b[0m:              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mst-finetuned-models-gamma'\u001b[0m, \u001b[38;2;0;135;0m'source_model_package_arn'\u001b[0m:          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28'\u001b[0m, \u001b[38;2;0;135;0m'base_model_arn'\u001b[0m:                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0'\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3_output_path'\u001b[0m: \u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval'\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'dataset_artifact_arn'\u001b[0m:                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m3138877d772ec489bef'\u001b[0m, \u001b[38;2;0;135;0m'action_arn_prefix'\u001b[0m:                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:action'\u001b[0m,               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'dataset_uri'\u001b[0m:                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-datas\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0met/gen_qa.jsonl'\u001b[0m, \u001b[38;2;0;135;0m'judge_model_id'\u001b[0m:                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'anthropic.claude-3-5-haiku-20241022-v1:0'\u001b[0m, \u001b[38;2;0;135;0m'llmaj_metrics'\u001b[0m:     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'\u001b[0m\u001b[1;38;2;0;135;0m[\u001b[0m\u001b[38;2;0;135;0m\"Completeness\", \"Faithfulness\"\u001b[0m\u001b[1;38;2;0;135;0m]\u001b[0m\u001b[38;2;0;135;0m'\u001b[0m, \u001b[38;2;0;135;0m'custom_metrics_s3_path'\u001b[0m:    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/evaluationinputs/eval-meta-\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m04295d9020251130-002201/custom-metrics.json'\u001b[0m, \u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m:  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'8192'\u001b[0m, \u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m, \u001b[38;2;0;135;0m'top_k'\u001b[0m: \u001b[38;2;0;135;0m'-1'\u001b[0m, \u001b[38;2;0;135;0m'top_p'\u001b[0m: \u001b[38;2;0;135;0m'1.0'\u001b[0m,       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'pipeline_name'\u001b[0m: \u001b[38;2;0;135;0m'SagemakerModelEvaluationType2-llmaj'\u001b[0m,          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'evaluate_base_model'\u001b[0m: \u001b[3;38;2;215;0;0mFalse\u001b[0m\u001b[1m}\u001b[0m                                    \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Rendered pipeline definition:                                    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#702\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">702</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"font-weight: bold\">{</span>                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Version\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"2020-12-01\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Metadata\"</span>: <span style=\"font-weight: bold\">{}</span>,                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"MlflowConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"color: #008700; text-decoration-color: #008700\">\"MlflowResourceArn\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">/mmlu-eval-experiment\"</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"font-weight: bold\">}</span>,                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Parameters\"</span>: <span style=\"font-weight: bold\">[]</span>,                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps\"</span>: <span style=\"font-weight: bold\">[</span>                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Lineage\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Actions\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ActionName\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ActionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>:                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackage\"</span>                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Properties\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineExecutionArn\"</span>: <span style=\"font-weight: bold\">{</span>                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionArn\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineName\"</span>:                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"SagemakerModelEvaluationType2-llmaj\"</span>                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Contexts\"</span>: <span style=\"font-weight: bold\">[</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ContextName\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ContextType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineExecution\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionArn\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Associations\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Action\"</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Context\"</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>:                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">3138877d772ec489bef\"</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluateCustomInferenceModel\"</span>,                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Training\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"TrainingJobName\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"CustomInference\"</span>,                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"RoleArn\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:iam::052150106756:role/Admin\"</span>,       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ServerlessJobConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"BaseModelArn\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"AcceptEula\"</span>: true,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"JobType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"BenchmarkEvaluation\"</span>                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"StoppingCondition\"</span>: <span style=\"font-weight: bold\">{</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"MaxRuntimeInSeconds\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">86400</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"HyperParameters\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"CustomInference\"</span>,                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"task\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"inference_only\"</span>                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"OutputDataConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"S3OutputPath\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval\"</span>, <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"CompressionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"NONE\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageGroupArn\"</span>:                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">st-finetuned-models-gamma\"</span>,                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceModelPackageArn\"</span>:                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"InputDataConfig\"</span>: <span style=\"font-weight: bold\">[</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ChannelName\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"train\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Prefix\"</span>,                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Uri\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-datas</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">et/gen_qa.jsonl\"</span>                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>,                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"DependsOn\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">]</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluateCustomModelMetrics\"</span>,                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Training\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"DependsOn\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluateCustomInferenceModel\"</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">]</span>,                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"TrainingJobName\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-llmaj-eval\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">{</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">]</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"RoleArn\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:iam::052150106756:role/Admin\"</span>,       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ServerlessJobConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"BaseModelArn\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"AcceptEula\"</span>: true,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"JobType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"LLMAJEvaluation\"</span>                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"StoppingCondition\"</span>: <span style=\"font-weight: bold\">{</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"MaxRuntimeInSeconds\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">86400</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"HyperParameters\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"name\"</span>: <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-llmaj-eval\"</span>,                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">{</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">]</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"judge_model_id\"</span>:                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"anthropic.claude-3-5-haiku-20241022-v1:0\"</span>,                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"inference_data_s3_path\"</span>: <span style=\"font-weight: bold\">{</span>                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"\"</span>,                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">{</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateCustomInferenceModel.OutputDataConfig.S3OutputPat</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">h\"</span>                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>,                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">{</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateCustomInferenceModel.TrainingJobName\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>,                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"/output/output/\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"CustomInference\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"/eval_results/inference_output.jsonl\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">]</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"output_path\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval\"</span>,  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"llmaj_metrics\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">[</span><span style=\"color: #008700; text-decoration-color: #008700\">\\\"Completeness\\\", </span>                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\\\"Faithfulness\\\"</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">]</span><span style=\"color: #008700; text-decoration-color: #008700\">\"</span>,                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"custom_metrics_s3_path\"</span>:                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval/evaluationinputs/eval-meta-</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">04295d9020251130-002201/custom-metrics.json\"</span>,                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_new_tokens\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"8192\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"temperature\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"0\"</span>,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_k\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-1\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_p\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"1.0\"</span>                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"OutputDataConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"S3OutputPath\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval\"</span>, <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"CompressionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"NONE\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageGroupArn\"</span>:                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">st-finetuned-models-gamma\"</span>,                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceModelPackageArn\"</span>:                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociateLineage\"</span>,                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Lineage\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"DependsOn\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">]</span>,                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifacts\"</span>: <span style=\"font-weight: bold\">[</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactName\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">}</span>,                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-inference-results\"</span>                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">]</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"InferenceResults\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateCustomInferenceModel.OutputDataConfig.S3OutputPat</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">h\"</span>                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactName\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">}</span>,                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-eval-report\"</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">]</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationReport\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateCustomModelMetrics.OutputDataConfig.S3OutputPath\"</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Associations\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>,                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-inference-results\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifact\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>,                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-eval-report\"</span>                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifact\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"font-weight: bold\">]</span>                                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"font-weight: bold\">}</span>                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Rendered pipeline definition:                                    \u001b]8;id=358999;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=565177;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#702\u001b\\\u001b[2m702\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1m{\u001b[0m                                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Version\"\u001b[0m: \u001b[38;2;0;135;0m\"2020-12-01\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Metadata\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m,                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"MlflowConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[38;2;0;135;0m\"MlflowResourceArn\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m/mmlu-eval-experiment\"\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[1m}\u001b[0m,                                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Parameters\"\u001b[0m: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m,                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Steps\"\u001b[0m: \u001b[1m[\u001b[0m                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Lineage\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Actions\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ActionName\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ActionType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m:                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceType\"\u001b[0m: \u001b[38;2;0;135;0m\"ModelPackage\"\u001b[0m                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Properties\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"PipelineExecutionArn\"\u001b[0m: \u001b[1m{\u001b[0m                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionArn\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"PipelineName\"\u001b[0m:                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"SagemakerModelEvaluationType2-llmaj\"\u001b[0m                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Contexts\"\u001b[0m: \u001b[1m[\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ContextName\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ContextType\"\u001b[0m: \u001b[38;2;0;135;0m\"PipelineExecution\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionArn\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Associations\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Action\"\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Context\"\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m:                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m3138877d772ec489bef\"\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluateCustomInferenceModel\"\u001b[0m,                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Training\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"TrainingJobName\"\u001b[0m: \u001b[38;2;0;135;0m\"CustomInference\"\u001b[0m,                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"RoleArn\"\u001b[0m: \u001b[38;2;0;135;0m\"arn:aws:iam::052150106756:role/Admin\"\u001b[0m,       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ServerlessJobConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"BaseModelArn\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"AcceptEula\"\u001b[0m: true,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"JobType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"EvaluationType\"\u001b[0m: \u001b[38;2;0;135;0m\"BenchmarkEvaluation\"\u001b[0m                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"StoppingCondition\"\u001b[0m: \u001b[1m{\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"MaxRuntimeInSeconds\"\u001b[0m: \u001b[1;36m86400\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"HyperParameters\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"name\"\u001b[0m: \u001b[38;2;0;135;0m\"CustomInference\"\u001b[0m,                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"task\"\u001b[0m: \u001b[38;2;0;135;0m\"inference_only\"\u001b[0m                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"OutputDataConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"S3OutputPath\"\u001b[0m: \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval\"\u001b[0m, \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"CompressionType\"\u001b[0m: \u001b[38;2;0;135;0m\"NONE\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ModelPackageConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"ModelPackageGroupArn\"\u001b[0m:                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mst-finetuned-models-gamma\"\u001b[0m,                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"SourceModelPackageArn\"\u001b[0m:                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"InputDataConfig\"\u001b[0m: \u001b[1m[\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ChannelName\"\u001b[0m: \u001b[38;2;0;135;0m\"train\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"S3DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3DataType\"\u001b[0m: \u001b[38;2;0;135;0m\"S3Prefix\"\u001b[0m,                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3Uri\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://my-sagemaker-sherpa-dataset/dataset/gen-qa-formatted-datas\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0met/gen_qa.jsonl\"\u001b[0m                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m,                                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"DependsOn\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m]\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluateCustomModelMetrics\"\u001b[0m,                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Training\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"DependsOn\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"EvaluateCustomInferenceModel\"\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m]\u001b[0m,                                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"TrainingJobName\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"custom-llmaj-eval\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m{\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m]\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"RoleArn\"\u001b[0m: \u001b[38;2;0;135;0m\"arn:aws:iam::052150106756:role/Admin\"\u001b[0m,       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ServerlessJobConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"BaseModelArn\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"AcceptEula\"\u001b[0m: true,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"JobType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"EvaluationType\"\u001b[0m: \u001b[38;2;0;135;0m\"LLMAJEvaluation\"\u001b[0m                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"StoppingCondition\"\u001b[0m: \u001b[1m{\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"MaxRuntimeInSeconds\"\u001b[0m: \u001b[1;36m86400\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"HyperParameters\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"name\"\u001b[0m: \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"custom-llmaj-eval\"\u001b[0m,                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m{\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m]\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"judge_model_id\"\u001b[0m:                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"anthropic.claude-3-5-haiku-20241022-v1:0\"\u001b[0m,                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"inference_data_s3_path\"\u001b[0m: \u001b[1m{\u001b[0m                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"\"\u001b[0m,                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m{\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateCustomInferenceModel.OutputDataConfig.S3OutputPat\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mh\"\u001b[0m                                                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m,                                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m{\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateCustomInferenceModel.TrainingJobName\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m,                                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"/output/output/\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"CustomInference\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"/eval_results/inference_output.jsonl\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m]\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"output_path\"\u001b[0m: \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval\"\u001b[0m,  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"llmaj_metrics\"\u001b[0m: \u001b[38;2;0;135;0m\"\u001b[0m\u001b[1;38;2;0;135;0m[\u001b[0m\u001b[38;2;0;135;0m\\\"Completeness\\\", \u001b[0m                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\\\"Faithfulness\\\"\u001b[0m\u001b[1;38;2;0;135;0m]\u001b[0m\u001b[38;2;0;135;0m\"\u001b[0m,                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"custom_metrics_s3_path\"\u001b[0m:                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval/evaluationinputs/eval-meta-\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m04295d9020251130-002201/custom-metrics.json\"\u001b[0m,                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_new_tokens\"\u001b[0m: \u001b[38;2;0;135;0m\"8192\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"temperature\"\u001b[0m: \u001b[38;2;0;135;0m\"0\"\u001b[0m,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_k\"\u001b[0m: \u001b[38;2;0;135;0m\"-1\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_p\"\u001b[0m: \u001b[38;2;0;135;0m\"1.0\"\u001b[0m                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"OutputDataConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"S3OutputPath\"\u001b[0m: \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval\"\u001b[0m, \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"CompressionType\"\u001b[0m: \u001b[38;2;0;135;0m\"NONE\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ModelPackageConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"ModelPackageGroupArn\"\u001b[0m:                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mst-finetuned-models-gamma\"\u001b[0m,                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"SourceModelPackageArn\"\u001b[0m:                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"AssociateLineage\"\u001b[0m,                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Lineage\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"DependsOn\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m]\u001b[0m,                                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Artifacts\"\u001b[0m: \u001b[1m[\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactName\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m}\u001b[0m,                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"custom-inference-results\"\u001b[0m                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m]\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactType\"\u001b[0m: \u001b[38;2;0;135;0m\"InferenceResults\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateCustomInferenceModel.OutputDataConfig.S3OutputPat\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mh\"\u001b[0m                                                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactName\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m}\u001b[0m,                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"custom-eval-report\"\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m]\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactType\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluationReport\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateCustomModelMetrics.OutputDataConfig.S3OutputPath\"\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Associations\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m,                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"custom-inference-results\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Artifact\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m,                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"custom-eval-report\"\u001b[0m                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Artifact\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[1m]\u001b[0m                                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1m}\u001b[0m                                                                \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:22:02] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found existing pipeline:                                              <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#199\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">199</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-LLMAJEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">f952b79f-4afe-4f2f-b45d-17894533c</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">6e9</span>                                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:22:02]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found existing pipeline:                                              \u001b]8;id=729179;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=511166;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#199\u001b\\\u001b[2m199\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-LLMAJEvaluation-\u001b[93mf952b79f-4afe-4f2f-b45d-17894533c\u001b[0m \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m6e9\u001b[0m                                                                   \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Updating pipeline                                                     <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#202\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">202</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-LLMAJEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">f952b79f-4afe-4f2f-b45d-17894533c</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">6e9</span> with latest definition                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Updating pipeline                                                     \u001b]8;id=567297;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=249002;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#202\u001b\\\u001b[2m202\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-LLMAJEvaluation-\u001b[93mf952b79f-4afe-4f2f-b45d-17894533c\u001b[0m \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m6e9\u001b[0m with latest definition                                            \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Updating pipeline resource.                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">resources.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30306\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">30306</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Updating pipeline resource.                                         \u001b]8;id=897054;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\u001b\\\u001b[2mresources.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=497721;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30306\u001b\\\u001b[2m30306\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:22:03] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Successfully updated pipeline:                                        <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#208\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">208</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-LLMAJEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">f952b79f-4afe-4f2f-b45d-17894533c</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">6e9</span>                                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:22:03]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Successfully updated pipeline:                                        \u001b]8;id=916795;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=385336;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#208\u001b\\\u001b[2m208\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-LLMAJEvaluation-\u001b[93mf952b79f-4afe-4f2f-b45d-17894533c\u001b[0m \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m6e9\u001b[0m                                                                   \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Starting pipeline execution: eval-meta-04295d90-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1764462123</span>            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#263\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">263</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Starting pipeline execution: eval-meta-04295d90-\u001b[1;36m1764462123\u001b[0m            \u001b]8;id=41189;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=464412;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#263\u001b\\\u001b[2m263\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Pipeline execution started:                                           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#274\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">274</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -LLMAJEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">f952b79f-4afe-4f2f-b45d-17894533c6e9</span>/execution/m318n <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         ngjk32f                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Pipeline execution started:                                           \u001b]8;id=227887;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=844359;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#274\u001b\\\u001b[2m274\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -LLMAJEvaluation-\u001b[93mf952b79f-4afe-4f2f-b45d-17894533c6e9\u001b[0m/execution/m318n \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         ngjk32f                                                               \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Evaluation job started!\n",
      "Job ARN: arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-LLMAJEvaluation-f952b79f-4afe-4f2f-b45d-17894533c6e9/execution/m318nngjk32f\n",
      "Job Name: eval-meta-04295d90\n",
      "Status: Executing\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">LLMAJEvaluationExecution</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">arn</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-LLMAJEvaluation-f952b79f-4afe-4f2f-b45d-17894533c6e9/execution/m318nngjk32f'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'eval-meta-04295d90'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[]</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">last_modified_time</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">datetime</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">.datetime</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2025</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">689000</span>, <span style=\"color: #d7af00; text-decoration-color: #d7af00\">tzinfo</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">tzlocal</span><span style=\"font-weight: bold\">())</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">eval_type</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">EvalType.LLM_AS_JUDGE:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #008700; text-decoration-color: #008700\">'llmasjudge'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">steps</span>=<span style=\"font-weight: bold\">[]</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mLLMAJEvaluationExecution\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0marn\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-LLMAJEvaluation-f952b79f-4afe-4f2f-b45d-17894533c6e9/execution/m318nngjk32f'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'eval-meta-04295d90'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m, \u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m, \u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mlast_modified_time\u001b[0m=\u001b[1;38;2;225;0;225mdatetime\u001b[0m\u001b[1;38;2;225;0;225m.datetime\u001b[0m\u001b[1m(\u001b[0m\u001b[1;36m2025\u001b[0m, \u001b[1;36m11\u001b[0m, \u001b[1;36m29\u001b[0m, \u001b[1;36m16\u001b[0m, \u001b[1;36m22\u001b[0m, \u001b[1;36m3\u001b[0m, \u001b[1;36m689000\u001b[0m, \u001b[38;2;215;175;0mtzinfo\u001b[0m=\u001b[1;38;2;225;0;225mtzlocal\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0meval_type\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225mEvalType.LLM_AS_JUDGE:\u001b[0m\u001b[39m \u001b[0m\u001b[38;2;0;135;0m'llmasjudge'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m=\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msteps\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run evaluation\n",
    "execution = evaluator.evaluate()\n",
    "\n",
    "print(f\"✅ Evaluation job started!\")\n",
    "print(f\"Job ARN: {execution.arn}\")\n",
    "print(f\"Job Name: {execution.name}\")\n",
    "print(f\"Status: {execution.status.overall_status}\")\n",
    "\n",
    "pprint(execution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Check Job Status\n",
    "\n",
    "Refresh and display the current job status with step details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'CreateEvaluationAction'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Starting'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T16:22:04.148000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&lt;sagemaker.core.utils.utils.Unassigned</span><span style=\"color: #008700; text-decoration-color: #008700\"> object at 0x1298e7170</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&gt;</span><span style=\"color: #008700; text-decoration-color: #008700\">'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'CreateEvaluationAction'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Starting'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T16:22:04.148000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'\u001b[0m\u001b[1;38;2;0;135;0m<\u001b[0m\u001b[1;38;2;0;135;0msagemaker.core.utils.utils.Unassigned\u001b[0m\u001b[38;2;0;135;0m object at 0x1298e7170\u001b[0m\u001b[1;38;2;0;135;0m>\u001b[0m\u001b[38;2;0;135;0m'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Refresh status\n",
    "execution.refresh()\n",
    "\n",
    "# Display job status using rich pprint\n",
    "pprint(execution.status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Monitor Pipeline Execution\n",
    "\n",
    "Poll the pipeline status until it reaches a terminal state (Succeeded, Failed, or Stopped)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Pipeline Execution Status</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Overall Status       </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Succeeded</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Target Status        </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Succeeded</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Elapsed Time         </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 1885.8s   </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pipeline Steps</span>                                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Step Name                       Status           Duration     </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> AssociateLineage               </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 1.9s         </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> EvaluateCustomModelMetrics     </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 1327.1s      </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> EvaluateCustomInferenceModel   </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 554.1s       </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> CreateEvaluationAction         </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 4.5s         </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34mPipeline Execution Status\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mOverall Status      \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[1;37mSucceeded\u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mTarget Status       \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[1;37mSucceeded\u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mElapsed Time        \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[37m1885.8s  \u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35mPipeline Steps\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mStep Name                     \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35mStatus         \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35mDuration    \u001b[0m\u001b[1;35m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mAssociateLineage              \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m1.9s        \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mEvaluateCustomModelMetrics    \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m1327.1s     \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mEvaluateCustomInferenceModel  \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m554.1s      \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mCreateEvaluationAction        \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m4.5s        \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:53:37] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Final Resource Status: Succeeded                                      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#979\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">979</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:53:37]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Final Resource Status: Succeeded                                      \u001b]8;id=524139;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=278480;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#979\u001b\\\u001b[2m979\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Wait for job completion (optional)\n",
    "# This will poll every 5 seconds for up to 1 hour\n",
    "execution.wait(poll=5, timeout=3600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 17:02:07] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted training job name:                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#52\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">52</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955 from   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         step: EvaluateCustomModelMetrics <span style=\"font-weight: bold\">(</span>priority: Custom<span style=\"font-weight: bold\">)</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 17:02:07]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted training job name:                                  \u001b]8;id=177834;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=168478;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#52\u001b\\\u001b[2m52\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955 from   \u001b[2m                        \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         step: EvaluateCustomModelMetrics \u001b[1m(\u001b[0mpriority: Custom\u001b[1m)\u001b[0m           \u001b[2m                        \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Result Artifacts Location</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">📦 </span><span style=\"font-weight: bold\">Full evaluation artifacts available at:</span>                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">  s3://mufi-test-serverless-smtj/eval/pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955/</span>                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34mResult Artifacts Location\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[1;34m📦 \u001b[0m\u001b[1mFull evaluation artifacts available at:\u001b[0m                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36m  s3://mufi-test-serverless-smtj/eval/pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955/\u001b[0m                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> S3 bucket: mufi-test-serverless-smtj, prefix: eval           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#341\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">341</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m S3 bucket: mufi-test-serverless-smtj, prefix: eval           \u001b]8;id=453165;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=425984;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#341\u001b\\\u001b[2m341\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted training job name:                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#52\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">52</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955 from   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         step: EvaluateCustomModelMetrics <span style=\"font-weight: bold\">(</span>priority: Custom<span style=\"font-weight: bold\">)</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted training job name:                                  \u001b]8;id=324161;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=683512;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#52\u001b\\\u001b[2m52\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955 from   \u001b[2m                        \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         step: EvaluateCustomModelMetrics \u001b[1m(\u001b[0mpriority: Custom\u001b[1m)\u001b[0m           \u001b[2m                        \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for bedrock summary in                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#361\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">361</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/pipelines-m318nngjk32f-E</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">valuateCustomModelM-lN73ONZ955/output/output/</span>                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for bedrock summary in                             \u001b]8;id=308182;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=660550;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#361\u001b\\\u001b[2m361\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/pipelines-m318nngjk32f-E\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mvaluateCustomModelM-lN73ONZ955/output/output/\u001b[0m                \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found bedrock job name: custom-llmaj-eval-m318nngjk32f       <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#377\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">377</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found bedrock job name: custom-llmaj-eval-m318nngjk32f       \u001b]8;id=705765;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=855376;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#377\u001b\\\u001b[2m377\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for JSONL in                                       <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#387\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">387</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/custom-llmaj-eval-m318nn</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">gjk32f/</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for JSONL in                                       \u001b]8;id=236968;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=874421;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#387\u001b\\\u001b[2m387\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/custom-llmaj-eval-m318nn\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mgjk32f/\u001b[0m                                                      \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found JSONL:                                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#405\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">405</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         eval/custom-llmaj-eval-m318nngjk32f/ld39q6di74sg/models/mode <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         l/taskTypes/General/datasets/CustomDataset/<span style=\"color: #ffff00; text-decoration-color: #ffff00\">4a22339b-b5b1-421</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">4-9c1e-0c0bf2c71fd6</span>_output.jsonl                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found JSONL:                                                 \u001b]8;id=648967;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=247115;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#405\u001b\\\u001b[2m405\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         eval/custom-llmaj-eval-m318nngjk32f/ld39q6di74sg/models/mode \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         l/taskTypes/General/datasets/CustomDataset/\u001b[93m4a22339b-b5b1-421\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m4-9c1e-0c0bf2c71fd6\u001b[0m_output.jsonl                             \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found results file:                                          <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#413\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">413</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         eval/custom-llmaj-eval-m318nngjk32f/ld39q6di74sg/models/mode <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         l/taskTypes/General/datasets/CustomDataset/<span style=\"color: #ffff00; text-decoration-color: #ffff00\">4a22339b-b5b1-421</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">4-9c1e-0c0bf2c71fd6</span>_output.jsonl                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found results file:                                          \u001b]8;id=234223;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=249361;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#413\u001b\\\u001b[2m413\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         eval/custom-llmaj-eval-m318nngjk32f/ld39q6di74sg/models/mode \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         l/taskTypes/General/datasets/CustomDataset/\u001b[93m4a22339b-b5b1-421\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m4-9c1e-0c0bf2c71fd6\u001b[0m_output.jsonl                             \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Loaded <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span> evaluation results                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#429\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">429</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Loaded \u001b[1;36m3\u001b[0m evaluation results                                  \u001b]8;id=139737;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=460642;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#429\u001b\\\u001b[2m429\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">═══ Evaluation 1 of 3 ═══</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1;36m═══ Evaluation 1 of 3 ═══\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Prompt:</span> What is the next number in this series? <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">16</span>, ?\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPrompt:\u001b[0m What is the next number in this series? \u001b[1;36m1\u001b[0m, \u001b[1;36m2\u001b[0m, \u001b[1;36m4\u001b[0m, \u001b[1;36m8\u001b[0m, \u001b[1;36m16\u001b[0m, ?\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model Response:</span> The next number in the series is <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel Response:\u001b[0m The next number in the series is \u001b[1;36m32\u001b[0m.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                               \n",
       " <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Metric                         </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">      Score </span> \n",
       " ───────────────────────────────────────────── \n",
       " <span style=\"color: #008080; text-decoration-color: #008080\"> Builtin.Completeness           </span> <span style=\"color: #008000; text-decoration-color: #008000\">     100.0% </span> \n",
       " <span style=\"color: #008080; text-decoration-color: #008080\"> Builtin.Faithfulness           </span> <span style=\"color: #008000; text-decoration-color: #008000\">     100.0% </span> \n",
       "                                               \n",
       "</pre>\n"
      ],
      "text/plain": [
       "                                               \n",
       " \u001b[1;35m \u001b[0m\u001b[1;35mMetric                        \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35m     Score\u001b[0m\u001b[1;35m \u001b[0m \n",
       " ───────────────────────────────────────────── \n",
       " \u001b[36m \u001b[0m\u001b[36mBuiltin.Completeness          \u001b[0m\u001b[36m \u001b[0m \u001b[32m \u001b[0m\u001b[32m    100.0%\u001b[0m\u001b[32m \u001b[0m \n",
       " \u001b[36m \u001b[0m\u001b[36mBuiltin.Faithfulness          \u001b[0m\u001b[36m \u001b[0m \u001b[32m \u001b[0m\u001b[32m    100.0%\u001b[0m\u001b[32m \u001b[0m \n",
       "                                               \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">═══ Evaluation 2 of 3 ═══</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1;36m═══ Evaluation 2 of 3 ═══\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Prompt:</span> What is the symbol that ends the sentence as a question\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPrompt:\u001b[0m What is the symbol that ends the sentence as a question\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model Response:</span> The symbol that ends the sentence as a question is: ?\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel Response:\u001b[0m The symbol that ends the sentence as a question is: ?\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                               \n",
       " <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Metric                         </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">      Score </span> \n",
       " ───────────────────────────────────────────── \n",
       " <span style=\"color: #008080; text-decoration-color: #008080\"> Builtin.Completeness           </span> <span style=\"color: #008000; text-decoration-color: #008000\">     100.0% </span> \n",
       " <span style=\"color: #008080; text-decoration-color: #008080\"> Builtin.Faithfulness           </span> <span style=\"color: #008000; text-decoration-color: #008000\">     100.0% </span> \n",
       "                                               \n",
       "</pre>\n"
      ],
      "text/plain": [
       "                                               \n",
       " \u001b[1;35m \u001b[0m\u001b[1;35mMetric                        \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35m     Score\u001b[0m\u001b[1;35m \u001b[0m \n",
       " ───────────────────────────────────────────── \n",
       " \u001b[36m \u001b[0m\u001b[36mBuiltin.Completeness          \u001b[0m\u001b[36m \u001b[0m \u001b[32m \u001b[0m\u001b[32m    100.0%\u001b[0m\u001b[32m \u001b[0m \n",
       " \u001b[36m \u001b[0m\u001b[36mBuiltin.Faithfulness          \u001b[0m\u001b[36m \u001b[0m \u001b[32m \u001b[0m\u001b[32m    100.0%\u001b[0m\u001b[32m \u001b[0m \n",
       "                                               \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">═══ Evaluation 3 of 3 ═══</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1;36m═══ Evaluation 3 of 3 ═══\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Prompt:</span> Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mPrompt:\u001b[0m Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model Response:</span> I ate a hamburger today and it was kind of dry.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel Response:\u001b[0m I ate a hamburger today and it was kind of dry.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                               \n",
       " <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Metric                         </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">      Score </span> \n",
       " ───────────────────────────────────────────── \n",
       " <span style=\"color: #008080; text-decoration-color: #008080\"> Builtin.Completeness           </span> <span style=\"color: #008000; text-decoration-color: #008000\">       0.0% </span> \n",
       " <span style=\"color: #008080; text-decoration-color: #008080\"> Builtin.Faithfulness           </span> <span style=\"color: #008000; text-decoration-color: #008000\">       0.0% </span> \n",
       "                                               \n",
       "</pre>\n"
      ],
      "text/plain": [
       "                                               \n",
       " \u001b[1;35m \u001b[0m\u001b[1;35mMetric                        \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35m     Score\u001b[0m\u001b[1;35m \u001b[0m \n",
       " ───────────────────────────────────────────── \n",
       " \u001b[36m \u001b[0m\u001b[36mBuiltin.Completeness          \u001b[0m\u001b[36m \u001b[0m \u001b[32m \u001b[0m\u001b[32m      0.0%\u001b[0m\u001b[32m \u001b[0m \n",
       " \u001b[36m \u001b[0m\u001b[36mBuiltin.Faithfulness          \u001b[0m\u001b[36m \u001b[0m \u001b[32m \u001b[0m\u001b[32m      0.0%\u001b[0m\u001b[32m \u001b[0m \n",
       "                                               \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">══════════════════════════════════════════════════════════════════════\n",
       "</pre>\n"
      ],
      "text/plain": [
       "══════════════════════════════════════════════════════════════════════\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">Showing evaluations </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> of </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36mShowing evaluations \u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;36m-\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;36m of \u001b[0m\u001b[1;36m3\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">══════════════════════════════════════════════════════════════════════\n",
       "</pre>\n"
      ],
      "text/plain": [
       "══════════════════════════════════════════════════════════════════════\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display results\n",
    "execution.show_results(limit=10, offset=0, show_explanations=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve an Existing Job\n",
    "\n",
    "You can retrieve and inspect any existing evaluation job using its ARN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 17:02:15] </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> Could not extract eval_type from ARN:                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#146\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">146</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -llmasjudge/execution/4hr7446yft1d                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 17:02:15]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m Could not extract eval_type from ARN:                                 \u001b]8;id=315627;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=953607;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#146\u001b\\\u001b[2m146\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -llmasjudge/execution/4hr7446yft1d                                    \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted s3_output_path from training job                            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">367</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-4hr7446yft1d-EvaluateCustomModelM-qePWbkcMxz:               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted s3_output_path from training job                            \u001b]8;id=739992;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=203397;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\u001b\\\u001b[2m367\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-4hr7446yft1d-EvaluateCustomModelM-qePWbkcMxz:               \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/\u001b[0m\u001b[38;2;225;0;225meval\u001b[0m                                   \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> Could not extract eval_type from ARN:                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#146\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">146</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -llmasjudge                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m Could not extract eval_type from ARN:                                 \u001b]8;id=550335;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=858100;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#146\u001b\\\u001b[2m146\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -llmasjudge                                                           \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> Could not extract eval_type from ARN:                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#146\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">146</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -llmasjudge/execution/4hr7446yft1d                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m Could not extract eval_type from ARN:                                 \u001b]8;id=379628;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=725705;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#146\u001b\\\u001b[2m146\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -llmasjudge/execution/4hr7446yft1d                                    \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'AssociateLineage'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:45:57.889000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:45:59.266000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateCustomModelMetrics'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:27:55.641000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:45:56.749000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateCustomInferenceModel'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:18:07.804000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:27:54.474000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'CreateEvaluationAction'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:18:05.550000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-19T15:18:07.332000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'AssociateLineage'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:45:57.889000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:45:59.266000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'EvaluateCustomModelMetrics'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:27:55.641000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:45:56.749000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'EvaluateCustomInferenceModel'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:18:07.804000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:27:54.474000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'CreateEvaluationAction'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:18:05.550000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-19T15:18:07.332000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #ff0000; text-decoration-color: #ff0000\">╭─────────────────────────────── </span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">Traceback </span><span style=\"color: #ff7f7f; text-decoration-color: #ff7f7f; font-weight: bold\">(most recent call last)</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\"> ────────────────────────────────╮</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">&lt;module&gt;</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">17</span>                                                                                   <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">14 </span>)                                                                                           <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">15 </span>pprint(existing_execution.status)                                                           <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">16 </span>                                                                                            <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>17 <span style=\"font-weight: bold; text-decoration: underline\">existing_execution.show_results</span>(limit=<span style=\"color: #0000ff; text-decoration-color: #0000ff\">5</span>, offset=<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>, show_explanations=<span style=\"color: #0000ff; text-decoration-color: #0000ff\">False</span>)                 <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">18 </span>                                                                                            <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/pydantic/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">main</span> <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">1026</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">__getattr__</span>                                                                          <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>                                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1023 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">return</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">super</span>().<span style=\"color: #00ff00; text-decoration-color: #00ff00\">__getattribute__</span>(item)  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># Raises AttributeError i</span>  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1024 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                 <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1025 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># this is the current error</span>                                       <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>1026 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff; font-weight: bold; text-decoration: underline\">raise</span><span style=\"font-weight: bold; text-decoration: underline\"> </span><span style=\"color: #00ffff; text-decoration-color: #00ffff; font-weight: bold; text-decoration: underline\">AttributeError</span><span style=\"font-weight: bold; text-decoration: underline\">(</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; text-decoration: underline\">f'{</span><span style=\"color: #00ffff; text-decoration-color: #00ffff; font-weight: bold; text-decoration: underline\">type</span><span style=\"font-weight: bold; text-decoration: underline\">(</span><span style=\"color: #00ffff; text-decoration-color: #00ffff; font-weight: bold; text-decoration: underline\">self</span><span style=\"font-weight: bold; text-decoration: underline\">).</span><span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold; text-decoration: underline\">__name__</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold; text-decoration: underline\">!r} object has no att</span>  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1027 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>                                                                                  <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1028 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">def</span><span style=\"color: #808080; text-decoration-color: #808080\"> </span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">__setattr__</span>(<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>, name: <span style=\"color: #00ffff; text-decoration-color: #00ffff\">str</span>, value: Any) -&gt; <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                             <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1029 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> (setattr_handler := <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.__pydantic_setattr_handlers__.get(name)) <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span>   <span style=\"color: #ff0000; text-decoration-color: #ff0000\">│</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000\">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">AttributeError: </span><span style=\"color: #008700; text-decoration-color: #008700\">'EvaluationPipelineExecution'</span> object has no attribute <span style=\"color: #008700; text-decoration-color: #008700\">'show_results'</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[38;2;255;0;0m╭─\u001b[0m\u001b[38;2;255;0;0m──────────────────────────────\u001b[0m\u001b[38;2;255;0;0m \u001b[0m\u001b[1;38;2;255;0;0mTraceback \u001b[0m\u001b[1;2;38;2;255;0;0m(most recent call last)\u001b[0m\u001b[38;2;255;0;0m \u001b[0m\u001b[38;2;255;0;0m───────────────────────────────\u001b[0m\u001b[38;2;255;0;0m─╮\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m in \u001b[92m<module>\u001b[0m:\u001b[94m17\u001b[0m                                                                                   \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m14 \u001b[0m)                                                                                           \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m15 \u001b[0mpprint(existing_execution.status)                                                           \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m16 \u001b[0m                                                                                            \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[31m❱ \u001b[0m17 \u001b[1;4mexisting_execution.show_results\u001b[0m(limit=\u001b[94m5\u001b[0m, offset=\u001b[94m0\u001b[0m, show_explanations=\u001b[94mFalse\u001b[0m)                 \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m18 \u001b[0m                                                                                            \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[2;33m/Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/pydantic/\u001b[0m\u001b[1;33mmain\u001b[0m \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[1;33m.py\u001b[0m:\u001b[94m1026\u001b[0m in \u001b[92m__getattr__\u001b[0m                                                                          \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m                                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1023 \u001b[0m\u001b[2m│   │   │   │   │   │   \u001b[0m\u001b[94mreturn\u001b[0m \u001b[96msuper\u001b[0m().\u001b[92m__getattribute__\u001b[0m(item)  \u001b[2m# Raises AttributeError i\u001b[0m  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1024 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                 \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1025 \u001b[0m\u001b[2m│   │   │   │   │   │   \u001b[0m\u001b[2m# this is the current error\u001b[0m                                       \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m \u001b[31m❱ \u001b[0m1026 \u001b[2m│   │   │   │   │   │   \u001b[0m\u001b[1;4;94mraise\u001b[0m\u001b[1;4m \u001b[0m\u001b[1;4;96mAttributeError\u001b[0m\u001b[1;4m(\u001b[0m\u001b[1;4;33mf\u001b[0m\u001b[1;4;33m'\u001b[0m\u001b[1;4;33m{\u001b[0m\u001b[1;4;96mtype\u001b[0m\u001b[1;4m(\u001b[0m\u001b[1;4;96mself\u001b[0m\u001b[1;4m).\u001b[0m\u001b[1;4;91m__name__\u001b[0m\u001b[1;4;33m!r}\u001b[0m\u001b[1;4;33m object has no att\u001b[0m  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1027 \u001b[0m\u001b[2m│   │   \u001b[0m                                                                                  \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1028 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mdef\u001b[0m\u001b[90m \u001b[0m\u001b[92m__setattr__\u001b[0m(\u001b[96mself\u001b[0m, name: \u001b[96mstr\u001b[0m, value: Any) -> \u001b[94mNone\u001b[0m:                             \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m│\u001b[0m   \u001b[2m1029 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m (setattr_handler := \u001b[96mself\u001b[0m.__pydantic_setattr_handlers__.get(name)) \u001b[95mis\u001b[0m \u001b[95mnot\u001b[0m   \u001b[38;2;255;0;0m│\u001b[0m\n",
       "\u001b[38;2;255;0;0m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
       "\u001b[1;91mAttributeError: \u001b[0m\u001b[38;2;0;135;0m'EvaluationPipelineExecution'\u001b[0m object has no attribute \u001b[38;2;0;135;0m'show_results'\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get an existing job by ARN\n",
    "# Replace with your actual pipeline execution ARN\n",
    "existing_arn = 'arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-llmasjudge/execution/4hr7446yft1d'  # or use a specific ARN\n",
    "\n",
    "from sagemaker.train.evaluate import EvaluationPipelineExecution\n",
    "from rich.pretty import pprint\n",
    "\n",
    "existing_execution = EvaluationPipelineExecution.get(\n",
    "    arn=existing_arn,\n",
    "    region=\"us-west-2\"\n",
    ")\n",
    "pprint(existing_execution.status)\n",
    "\n",
    "existing_execution.show_results(limit=5, offset=0, show_explanations=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get All LLM-as-Judge Evaluations\n",
    "\n",
    "Retrieve all LLM-as-Judge evaluation jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 17:02:21] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted s3_output_path from training job                            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">367</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955:               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 17:02:21]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted s3_output_path from training job                            \u001b]8;id=802368;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=75226;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#367\u001b\\\u001b[2m367\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-m318nngjk32f-EvaluateCustomModelM-lN73ONZ955:               \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/\u001b[0m\u001b[38;2;225;0;225meval\u001b[0m                                   \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 LLM-as-Judge evaluation jobs\n",
      "  - m318nngjk32f: Succeeded\n",
      "  - 2m5hczli7vdp: Failed\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.train.evaluate import LLMAsJudgeEvaluator\n",
    "\n",
    "# Get all LLM-as-Judge evaluations as an iterator\n",
    "all_executions = list(LLMAsJudgeEvaluator.get_all(region=\"us-west-2\"))\n",
    "\n",
    "print(f\"Found {len(all_executions)} LLM-as-Judge evaluation jobs\")\n",
    "for execution in all_executions:\n",
    "    print(f\"  - {execution.name}: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop a Running Job (Optional)\n",
    "\n",
    "If needed, you can stop a running evaluation job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to stop the job\n",
    "# execution.stop()\n",
    "# print(f\"Execution stopped. Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Support\n",
    "\n",
    "The `dataset` parameter supports two formats:\n",
    "\n",
    "### 1. S3 URI\n",
    "```python\n",
    "dataset=\"s3://my-bucket/path/to/dataset.jsonl\"\n",
    "```\n",
    "\n",
    "### 2. Dataset ARN (AI Registry)\n",
    "```python\n",
    "dataset=\"arn:aws:sagemaker:us-west-2:123456789012:hub-content/AIRegistry/DataSet/my-dataset/1.0.0\"\n",
    "```\n",
    "\n",
    "The evaluator automatically detects which format is provided and uses the appropriate data source configuration."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
